Skip to content

ENH: Replace the shell wheel build system with Python (supersedes #302) - #309

Draft
hjmjohnson wants to merge 131 commits into
InsightSoftwareConsortium:mainfrom
BRAINSia:itk6-build-system-v2
Draft

hjmjohnson wants to merge 131 commits into
InsightSoftwareConsortium:mainfrom
BRAINSia:itk6-build-system-v2

Conversation

@hjmjohnson

@hjmjohnson hjmjohnson commented Sep 17, 2026 •

Copy link
Copy Markdown
Member

Replaces the shell and PowerShell wheel-build system with a pixi-driven
Python one, and moves ITK 6 to stable-ABI (abi3) wheels on
manylinux_2_28, macOS arm64 14.0, and Windows x86_64. Supersedes #302.

Linux and macOS wheels now build with --dead-strip by default (port of
SimpleITK/SimpleITK#2784): itk_core shrinks from 89.5 to 71.4 MB on Linux
and from 61.5 to 49.3 MB on macOS, with unchanged outputs.

Successfully built on Windows, macOS arm64, Linux x86_64, and Linux
aarch64. The Doxygen/environment defect found by @dzenanz is fixed: build
tools now resolve inside the target --platform-env, and launching from
the wrong environment fails immediately with the environment name to use.

Build caches for all four platforms are published to ITKPythonBuilds
v6.0rc01.dev20260915; the macOS cache targets macOS 14.0. Those caches
and the platform builds below predate the eighth package,
itk-filtering-extra; the current tree has been built only on Linux
x86_64 (all 8 packages, see the --dead-strip measurements).

Release decisions for reviewers: a new PyPI package, itk-filtering-extra
(the filtering modules ingested from remote modules), and itk-io now
requires itk-numerics.

Platform verification — clean builds from this branch

Every row is a from-scratch build of this branch, not an incremental one,
made before itk-filtering-extra was added (hence 7 wheels).

Platform Result Time Wheels
Linux x86_64 pass — 7 × cp311-abi3-manylinux_2_28_x86_64
Linux aarch64 pass on retry 39 min 7 × cp311-abi3-manylinux_2_28_aarch64
macOS arm64 pass 93 min 7 × cp311-abi3-macosx_14_0_arm64
Windows x86_64 pass — 7 × cp311-abi3-win_amd64

macOS binaries were checked with otool -l: real .so files report
minos 14.0, matching their macosx_14_0_arm64 tag. That check matters
because a component wheel tagged below its true minos installs on older
macOS and then fails in dyld at import — installable and broken, rather
than loudly uninstallable.

Remote-module end-to-end (the consumer path)

Proven on Windows x86_64 and Linux x86_64, each run as a fresh
consumer with zero ITK wheels present beforehand, so a leftover
artifact could not produce a false pass.

Linux x86_64, ITKBioCell, 8 minutes, rc=0:

  • itk_biocell-1.0.0-cp311-abi3-manylinux_2_28_x86_64.whl, 0.58 MB
  • Requires-Dist: itk<7,>=6.0.0b1 — the dependency rewrite fired
  • 2 bundled libraries — genuinely repaired by auditwheel
  • pyproject.toml restored byte-identical after the build

An earlier Windows pass was found to be riding on leftover wheels; the
zero-wheel precondition was added in response and both platforms were
re-run under it.

--dead-strip: measurements

Linux links with -ffunction-sections -fdata-sections, --gc-sections,
and a version script that exports only PyInit_* from each Python module.
--no-dead-strip opts out. User -D flags such as
-DCMAKE_CXX_FLAGS:STRING=... are merged with these, not replaced.

Measure Before --dead-strip
_ITKCommonPython exported symbols 33,625 84
itk_core wheel, Linux x86_64 89.5 MB 71.4 MB
ITK Python tests (252) 244 pass 244 pass, same 8 failures
Registration benchmark outputs (17 cases) bit-identical

The export limit carries the saving; --gc-sections alone gave 3-5%.
Linux figures are from the pushed configuration (default linker, with a
user -DCMAKE_CXX_FLAGS:STRING=-pipe merged in). Linking with gold gave
wheels about 1% smaller (itk_core 70.7 MB) and was dropped. A remote
module (ITKSimpleITKFilters) built against the stripped wheels gives
identical output in all four old/new ITK and module combinations.

Windows needs no change: a .pyd already exports only PyInit_* (84 on
_ITKCommonPython), and Release links use /OPT:REF,ICF by default.
macOS Python modules link with -dead_strip and export only _PyInit_*:
wheels are 20-48% smaller (itk_core 61.5 to 49.3 MB), benchmark outputs
are identical, and BSpline LBFGSB registration runs 15% faster.
-dead_strip alone changed nothing (<0.1%).

The 8 ITK test failures occur with and without the change and come from
the wheel configuration (unwrapped return types such as
SinusoidSpatialFunctionF4PD4, TransformToStrainFilter for
Transform[D,3,3], itk.Ransac) or missing test data
(Real-DSA-01-Speed-02.mhd).

Unit tests and hooks
  • pytest: 205 passed. pytest is a developer-only dependency; the
    normal build path does not import it.
  • pre-commit run --all-files: exit 0, every hook Passed
    (black, ruff, shellcheck, shfmt, taplo, yaml/toml, whitespace).
Known caveat: intermittent aarch64 link truncation

One clean aarch64 build failed while linking:

FAILED: lib/libITKIOStimulate-6.0.a
ranlib: error reading itkStimulateImageIO.cxx.o: file truncated

The identical clean build then passed, with no truncation and no short
object files. Evidence points at the container file-sharing layer rather
than this branch: 158 GB free at the time, the truncated archive was
exactly 49,152 bytes (12 × 4096, a dropped write tail rather than a
partial compile), and aarch64 is the only platform writing through a
Docker bind mount onto a macOS filesystem.

Docker Desktop updated itself between the two runs (29.7.2 → 29.8.0), so
the version cannot be credited with the fix, and one pass does not
disprove an intermittent fault. Recommended handling: retry a clean
build first; do not treat a truncation as a defect in this branch.

Commit series

Topical commits, no fixups and no merge commits, now including
@dzenanz's wheels-6.0b3 series and the --dead-strip commits. Some
BUG: commits fix earlier commits in this series; they are kept separate
because they record distinct defects found in testing.

Beyond the build-system replacement, the series also adds:

  • release-wheels, one command per release build machine
  • the ITK modules ingested from remote modules, built into the wheels
  • itk-filtering-extra, a new package for the ingested filtering modules
  • itk-io depending on itk-numerics
  • a build failure when a wheel exceeds the 100 MB PyPI limit
  • building a remote module against an installed ITK (--module-itk-dir install)
  • ninja jobs bounded by physical memory (ITK_BUILD_JOBS overrides)

Cavan Riley is credited via Co-Authored-By on the seven commits
carrying his original work.

@dzenanz dzenanz left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have taken a look. There is too much code to look carefully. I tried running locally:

M:\a\IPP>pixi run python3 "C:\Misc\ITKPythonPackage\scripts\build_wheels.py" --platform-env windows-py311 --itk-source-dir C:/Dev/ITK-git
Error:   × could not find pixi.toml or pyproject.toml with tool.pixi at directory M:\a\IPP


M:\a\IPP>cd C:\Misc\ITKPythonPackage

M:\a\IPP>c:

C:\Misc\ITKPythonPackage>pixi run python3 "C:\Misc\ITKPythonPackage\scripts\build_wheels.py" --platform-env windows-py311 --itk-source-dir C:/Dev/ITK-git
 WARN Encountered 2 warnings while parsing the manifest:
  ⚠ the `[system-requirements]` table is deprecated in favor of virtual packages on `platforms`
    ╭─[C:\Misc\ITKPythonPackage\pixi.toml:82:1]
 81 │
 82 │ ╭─▶ [feature.manylinux228-build.system-requirements]
 83 │ │   linux = "5.4"
 84 │ ├─▶ libc = { family = "glibc", version = "2.28" }
    · ╰──── declare these on the `platforms` entries instead
 85 │
    ╰────
  help: e.g. platforms = [{ platform = "linux-64", cuda = "12" }]

  ⚠ the `[system-requirements]` table is deprecated in favor of virtual packages on `platforms`
    ╭─[C:\Misc\ITKPythonPackage\pixi.toml:95:23]
 94 │ platforms = ["osx-arm64"]
 95 │ system-requirements = { macos = "14.0" }
    ·                       ─────────┬────────
    ·                                ╰── declare these on the `platforms` entries instead
 96 │
    ╰────
  help: e.g. platforms = [{ platform = "linux-64", cuda = "12" }]

Error launching 'python3': The system cannot find the path specified. (os error 3)

C:\Misc\ITKPythonPackage>pixi run python "C:\Misc\ITKPythonPackage\scripts\build_wheels.py" --platform-env windows-py311 --itk-source-dir C:/Dev/ITK-git
 WARN Encountered 2 warnings while parsing the manifest:
  ⚠ the `[system-requirements]` table is deprecated in favor of virtual packages on `platforms`
    ╭─[C:\Misc\ITKPythonPackage\pixi.toml:82:1]
 81 │
 82 │ ╭─▶ [feature.manylinux228-build.system-requirements]
 83 │ │   linux = "5.4"
 84 │ ├─▶ libc = { family = "glibc", version = "2.28" }
    · ╰──── declare these on the `platforms` entries instead
 85 │
    ╰────
  help: e.g. platforms = [{ platform = "linux-64", cuda = "12" }]

  ⚠ the `[system-requirements]` table is deprecated in favor of virtual packages on `platforms`
    ╭─[C:\Misc\ITKPythonPackage\pixi.toml:95:23]
 94 │ platforms = ["osx-arm64"]
 95 │ system-requirements = { macos = "14.0" }
    ·                       ─────────┬────────
    ·                                ╰── declare these on the `platforms` entries instead
 96 │
    ╰────
  help: e.g. platforms = [{ platform = "linux-64", cuda = "12" }]

Running >>>>>: git fetch --tags origin  ; # in cwd=C:\Dev\ITK-git with check=False

Traceback (most recent call last):
  File "C:\Misc\ITKPythonPackage\scripts\build_wheels.py", line 628, in <module>
    build_wheels_main()
    ~~~~~~~~~~~~~~~~~^^
  File "C:\Misc\ITKPythonPackage\scripts\build_wheels.py", line 450, in build_wheels_main
    raise RuntimeError(
    ...<2 lines>...
    )
RuntimeError: Failed to fetch ITK tags in C:\Dev\ITK-git: fatal: 'origin' does not appear to be a git repository
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.


C:\Misc\ITKPythonPackage>

@dzenanz

dzenanz commented Sep 18, 2026

Copy link
Copy Markdown
Member

This seems to be doing something: C:\Misc\ITKPythonPackage>pixi run python scripts/build_wheels.py --platform-env windows-py311 --itk-git-tag v6.0b01

@dzenanz

dzenanz commented Sep 18, 2026

Copy link
Copy Markdown
Member

That failed too: run.log

@hjmjohnson
hjmjohnson force-pushed the itk6-build-system-v2 branch 8 times, most recently from ef0c2d4 to 1af2b15 Compare September 19, 2026 20:08
@hjmjohnson

Copy link
Copy Markdown
Member Author

Windows verification complete — handing back to Linux/macOS

Everything Windows-specific on this branch is now verified by real builds, and one cross-platform defect found along the way is fixed. Head is 1af2b15 (10 commits, no BUG: commits — every fix is folded into the commit that introduced the code). Coordination detail lives in the shared gist; this is the summary a reviewer needs.

What was verified on Windows 11 / VS 2022 (i7-9800X, 32 GB)

step result
clean cache build from ef0c2d4 (pixi reinstalled, everything deleted first) 7 × cp311-abi3-win_amd64, cache 226,973,373 B, published to v6.0rc01.dev20260915, byte-identical
DLL bundling 9 DLLs in itk_core incl. itk_core.libs/tbb12.dll; wheels import in a clean venv with PATH stripped to System32 and ITK_EAGER_IMPORT=1
remote module (ITKBioCell) via the documented script, as a fresh consumer step 06 reached; Requires-Dist: itk<7,>=6.0.0b1; repaired wheel (with msvcp140.dll) in the module's own dist/; pyproject.toml restored byte-identical
unit tests 97 passed on Windows (was 11 failed / 81 passed)

Defects fixed on this branch by the Windows work

All Windows-only unless noted; none were reachable from Linux/macOS testing.

  1. TargetPlatform.detect() read PROCESSOR_ARCHITECTURE from the config mapping, not the environment — Unknown machine '' on any Windows build (POSIX hidden by os.uname).
  2. Nested pixi run from inside an activated env overflowed the cmd.exe 8191-char limit (The input line is too long) — in the interpreter probe and in echo_check_call.
  3. build_tarball() deleted the seven freshly built wheels from dist/ and reported rc=0.
  4. pixi's vs2022 activation exports CMAKE_GENERATOR="Visual Studio 17 2022"; scikit-build-core honoured it, and the multi-config generator broke ITK wrapping's file(GENERATE) (castxml.inc "written multiple times"). Module wheels now pass -GNinja.
  5. Cross-platform: the dependency rewrite emitted itk >= 6.0, < 7, which PEP 440 makes unsatisfiable by any 6.0.0bN (6.0.0b2 < 6.0); --pre cannot rescue it. Now >= 6.0.0b1, < 7.
  6. delvewheel --add-path pointed at a non-existent oneTBB directory.
  7. pixi global install aria2 always fails on win-64 and printed a red error that looked fatal.
  8. A failed git checkout <tag> forced git checkout main, moving a cache-restored ITK source off the commit its binaries were built from.
  9. 7-Zip -r treated build as a recursive pattern and swept a remote module's CMake tree into the cache; consumers then hit generator ... does not match.
  10. Windows fixup_wheel ignored remote_module_wheel and wrote the repaired module wheel to the main dist/, leaving the unrepaired one as the module's only artifact.
  11. Cross-platform (gist T7): on the --skip-itk-wheel-build path, 04_post_build_fixup and 05_final_import_test still ran against ITK wheels that were never built, and step 05 aborted the run before the module wheel. Both are now registered as _skipped no-ops when 03 is skipped; the step table was extracted into build_step_table() and is covered by tests. The earlier Windows "pass" of step 05 was an artifact of leftover wheels on the build machine — verified by reproducing the failure with dist/ empty, then re-running after the fix.

Plus three test fixes so the suite runs on Windows (os.uname monkeypatching, a cross-build test that inferred the host OS, a symlink that needs SeCreateSymbolicLinkPrivilege), and install-pixi.ps1 added to .gitignore.

For Linux/macOS

  • T2 (Linux remote-module test) is unblocked by fix 11 — git fetch, rebuild from 1af2b15 or later. Expected Requires-Dist is itk<7,>=6.0.0b1, not the older >= 6.0, < 7.
  • The published caches were built from ef0c2d4 and carry the pre-fix-11 scripts. The documented module scripts re-clone the branch when ITKPYTHONPACKAGE_TAG is set, so consumers get the fix; a consumer without that variable does not. Rebuilding caches is not urgent but is why cache and branch can disagree.
  • Not on this branch: an experiment bounding ninja jobs by physical memory (-j = min(cpu, ram/4) with a link pool). Motivated by two OOM kills at -j16 on 32 GB; peak memory dropped to 9.9 GB, but the timing run was confounded and is parked on a separate local branch. Worth revisiting for CI runner sizing.

@hjmjohnson
hjmjohnson force-pushed the itk6-build-system-v2 branch 2 times, most recently from c219d94 to afdfaf8 Compare September 20, 2026 13:19
@hjmjohnson

Copy link
Copy Markdown
Member Author

Thanks for running it locally — your run.log found a real defect in this
branch, and your two earlier comments are now answered by the Windows
verification. Rebased onto current main and pushed; head is afdfaf8.

Your build failure is our bug, not your invocation. The log ends at:

CMake Error: Could NOT find Doxygen (missing: DOXYGEN_EXECUTABLE)
  -DITK_WRAP_DOC:BOOL='ON'  -DDOXYGEN_EXECUTABLE:FILEPATH='C:\Dev\doxygen'

ITK_WRAP_DOC=ON is set unconditionally and DOXYGEN_EXECUTABLE comes from
which_required("doxygen"), which aborts when it cannot resolve. But
doxygen is not declared in pixi.toml, so it is an undeclared external
prerequisite. My Windows runs passed only because that machine happens to have
a working Doxygen. That defeats the point of using pixi for a hermetic
toolchain, and it is being fixed before this leaves draft.

310-x64 is removed (your inline comment). It was a leftover, and it could
never have resolved: the only environments in pixi.toml are dev, publish,
linux-py311, manylinux228-py311, macosx-py311, and windows-py311.
ITK 5.4 is out of scope for this build system.

This PR stays WIP until the Doxygen prerequisite is resolved. "Too much
code to look carefully" is fair — please don't spend review time on the whole
diff yet; I'll ping you when it is ready for a real pass.

Platform verification status as of afdfaf8
Platform Clean build Remote-module end-to-end
Linux x86_64 pass pass (ITKBioCell, 8 min)
Linux aarch64 pass on retry —
macOS arm64 pass (93 min) —
Windows x86_64 pass pass

97 unit tests pass; pre-commit run --all-files exits 0.

One known caveat: a single aarch64 clean build failed while linking with a
truncated object file, and the identical rebuild passed. Evidence points at the
Docker bind-mount layer rather than this branch (the truncated archive was
exactly 49,152 bytes, a dropped write tail, and aarch64 is the only platform
writing through a bind mount onto a macOS filesystem). Docker updated itself
between the two runs, so the version cannot be credited with the fix.

@hjmjohnson hjmjohnson changed the title ENH: Replace the shell wheel build system with Python (supersedes #302) WIP: ENH: Replace the shell wheel build system with Python (supersedes #302) Sep 20, 2026
@hjmjohnson

Copy link
Copy Markdown
Member Author

@dzenanz thank you for the log file — it identified a real bug, and that bug is
now fixed in e53db8d.

This should have been a WIP: PR from the start, and that is my fault. It
was opened primarily to get the code onto my Windows machine for testing and to
track the outstanding testing tasks, not to request review. It is now retitled
WIP:. There are still refinements to make, but I have successfully run it on
Windows, macOS arm64, Linux x86_64, and Linux aarch64 builds.

What your log found, and the fix

Your run failed at:

CMake Error: Could NOT find Doxygen (missing: DOXYGEN_EXECUTABLE)
  -DDOXYGEN_EXECUTABLE:FILEPATH='C:\Dev\doxygen'

doxygen is declared in pixi.toml and is present in every build
environment. The bug was that your command used pixi run without -e,
which lands in the default environment. The script then resolved its build
tools from whatever was on the system PATH rather than from the target
environment, and handed CMake a C:\Dev\doxygen that is not a Doxygen
executable. Reproduced here — in default, cmake, ninja, doxygen, and
git all resolve to system paths.

Two changes:

  1. cmake, ninja, doxygen, and git now resolve inside the target
    --platform-env
    , the same way the Python interpreter already did. A stray
    same-named executable on PATH can no longer reach CMake.
  2. Running outside the requested environment now fails immediately, before
    any work, naming the environment you want:
==============================================================
  WRONG ENVIRONMENT - build aborted before doing any work
==============================================================
  running in : the 'default' pixi environment
  required   : 'windows-py311'

  Re-run with the environment selected:
      pixi run -e windows-py311 python scripts/build_wheels.py ...

  'pixi run' without '-e' uses the 'default' environment, which
  does NOT carry the pinned cmake, ninja, doxygen and git. The
  build would silently pick up whatever is on the system PATH.
==============================================================

Six regression tests cover both (103 total, up from 97). Your exact command
shape now exits 1 with that message instead of failing minutes later inside
CMake.

@hjmjohnson
hjmjohnson force-pushed the itk6-build-system-v2 branch 2 times, most recently from fb3e45b to 42922e0 Compare September 20, 2026 14:10
@hjmjohnson
hjmjohnson force-pushed the itk6-build-system-v2 branch 3 times, most recently from 1af2b15 to ca9994a Compare September 20, 2026 14:22
ctypes structure fields are Any to mypy; the Windows memory probe now
converts ullTotalPhys to int before dividing.
Thirteen findings, none a behavior change. Four guards tested a
CMakeArgumentBuilder for truth, which is always true since it has no
__len__; they only skipped an update() that is a no-op when empty, so
they are removed. The remote-module path goes through
required_module_source_dir(), and paths added to command lists are
converted with str(), as echo_check_call did for them already.
Bare dict, list, Callable, CompletedProcess, Match, and PathLike get
their arguments. CompletedProcess[str] shows echo_check_call's bytes
branches were dead: run_commandLine_subprocess always runs with
text=True, so stdout and stderr are printed directly.
parameter_option now indexes its defaults, so an unknown option raises
KeyError where it used to return None into a string multiplication.
push_dir declares its attributes, which mypy had read as always None.
test_non_git_source_is_not_fetched and test_git_source_still_fetches
installed a fake git and never called build_wheels, so they passed
whatever the guard did. The fetch-and-checkout block moves into
sync_itk_source(), which the tests now drive through each outcome: no
repository, a clean sync, a fatal and a survivable fetch failure, and a
failed checkout. The source-text checks for the guard, the gitauto
default, and the cache-tree error message become calls to the code.
The definitions every ITK wrapping build is configured with move from
the builder's constructor into itk_source_build_configuration(), so a
test can read them without the CMake and pixi paths the constructor
resolves. The check for ITK_INSTALL_WRAPPING_DEVELOPMENT_FILES calls
it instead of searching the source text.
The check searched wheel_packaging.py for the exclusion flag. It now
builds a tree holding an ITK .git directory, runs the real tar command
create_posix_tarball produces, and confirms the archive keeps the
sources and holds no .git entry at all. Excluding only the directory's
contents fails it.
mypy's disallow_untyped_calls reported 61 calls into unannotated test
helpers and one into build_tarball(). The helpers now carry the builder
classes, Path, and str types they are actually used with;
_make_builder is generic over the builder class so each fixture keeps
its concrete type. build_tarball() returns None on every platform.

Any appears only in the Path.open stand-in in test_build_jobs.py: it
forwards whatever positional and keyword arguments the code under test
passes to open(), which are genuinely dynamic.
mypy inferred the recording dict in
test_install_step_runs_cmake_install_into_the_prefix as holding lists
only, from its first assignment, and so judged the `check` comparison
against True impossible. The dict holds a command list and a bool;
declaring it dict[str, object] states that, and the comparison is
checked as written.
Three tests reached the standard library through the module under test
(build_jobs.platform, build_jobs.os, pyproject_configure.sys), which
those modules never export. Each name is the same module object as the
test's own import, so patching platform, os, or sys directly replaces
exactly what the code under test sees.
Every flag strict enables now passes on its own, so the seven per-flag
lines give way to strict = true, which is what sp-repo-review MY101
asks for. Unannotated functions stay allowed, the staged path the
Scientific Python style guide describes: disallow_untyped_defs and
disallow_incomplete_defs are off, and check_untyped_defs still checks
their bodies. warn_unreachable and the three extra error codes are not
part of strict and stay.
A contributor without pixi needs one entry point to the checks CI runs.
noxfile.py wraps the existing commands rather than defining new ones:
lint runs pre-commit on every file, tests runs pytest with the
pyproject.toml settings against the dev dependency group on Python 3.11
to 3.14, and docs runs the sphinx-build line from the docs workflow
(off by default; --serve uses sphinx-autobuild).

The file follows the Scientific Python tasks guide: a uv script block
and shebang, a minimum nox version, uv with a virtualenv fallback, and
default= per session. Wheel builds stay pixi tasks because nox does not
manage compiled builds well.
Contributors need to know which sessions exist, which run by default,
how to pass arguments through, and that wheel builds remain pixi tasks.
compiler_cache.py picks the tool by platform and returns the launcher
definitions. ITK_USE_CCACHE is set explicitly: ITK turns it on whenever a
launcher is set and then runs CastXML and SWIG through the launcher,
which ccache caches and sccache rejects. The cache is sized at 30G unless
the caller set CCACHE_MAXSIZE or SCCACHE_CACHE_SIZE, since a wrapped ITK
build overflows ccache's 5 GB default.
The flag was combined with the environment by "or", so --no-use-ccache
could not turn off a cache that USE_CCACHE=1 asked for. An explicit flag
now wins and the environment decides only without one. The flag gains
the name --use-compiler-cache, since on Windows it means sccache;
--use-ccache stays as an alias, and USE_COMPILER_CACHE joins USE_CCACHE.
oneTBB, and ITK when the superbuild builds it, now compile through the
same cache. ITK_USE_CCACHE is forwarded with it, so the superbuild's ITK
does not default it on and put SWIG and CastXML behind sccache.
ccache 4.14 for the macOS, Linux, and manylinux builds, sccache 0.19 for
Windows. The lock adds them and their dependencies (libhiredis, xxhash)
and changes nothing else.
The container mounts a host directory as its CCACHE_DIR, one per
manylinux version and architecture, since ccache keys on the compiler
and the x64 runner and the root aarch64 container must not share
ownership of one directory. ITK_CONTAINER_CCACHE_DIR overrides the
location. USE_CCACHE passes through, and the ITK wheel build no longer
forces --no-use-ccache, so the host decides.
When a cache is used, its statistics are zeroed before the build steps
and shown after them, even when a step fails, so each daily build logs
its own hit rate. The sccache server keeps the environment it started
with, so it is restarted after the cache size is set and stopped at the
end. The report runs unchecked and cannot mask a build failure.
Daily releases rebuild mostly unchanged sources on every platform. Each
native release build passes --use-compiler-cache, and the manylinux
builds pass USE_CCACHE=1 into the container, which keeps its cache in a
host directory between runs. A cache hit returns the object the compiler
produced for the same source, compiler, and flags, so the wheels do not
change.
On some hosts the build container cannot resolve names through the
runtime's default resolver, so pixi and git fail inside it unless
each run is given a --dns workaround by hand. CONTAINER_DNS takes a
space- or comma-separated list of servers and adds one --dns per entry
to the container arguments, for both the dockcross runner (x64) and
the direct run (aarch64). Unset, the arguments are unchanged.

release_wheels.py runs each step with the caller's environment, so a
CONTAINER_DNS exported by the user reaches the script.

The module wheel script runs its container with --network=host, which
uses the host resolver and which docker refuses to combine with --dns,
so it is left unchanged.
The 30G limit was set only in the build's own environment, so any later
ccache call outside the build (ccache -s, a manual run) saw the 5 GB
default and could trim a warm cache to 5 GB on its next cleanup.

Run `ccache --max-size <size>` before the build so max_size is written to
the cache directory's ccache.conf. The size is the caller's CCACHE_MAXSIZE
when set, otherwise the default; the builder passes it in so
compiler_cache_session stays a pure function.

sccache is unchanged: it has no per-cache-directory setting. Its disk
cache size comes from SCCACHE_CACHE_SIZE or [cache.disk] size in the
per-user config file named by SCCACHE_CONF (docs/Configuration.md), and
it has no command to set a size.
Two release builds of the same ITK commit produced wheels whose every
member had the same contents, but the wheel files differed because each
zip entry carried the build time. scikit-build-core, wheel pack,
auditwheel, delocate, and delvewheel all take their zip timestamps from
SOURCE_DATE_EPOCH when it is set, so build_wheels.py now sets it to the
committer time of the ITK revision being built. A value the caller
already set is kept. A source tree without a repository (a tree restored
from a build cache) has no commit time; the variable is then left unset
with a warning rather than given an invented date, and git is stopped
at the tree's parent so it cannot read an enclosing repository's time.

The manylinux scripts forward a SOURCE_DATE_EPOCH set on the host into
the container, as they do for the compiler cache settings.
build_wheels.py reads the ITK commit time inside each build. In the
root-run aarch64 docker path git can refuse the mounted repository as
"dubious ownership", which leaves SOURCE_DATE_EPOCH unset and the
wheels' zip timestamps nondeterministic.

release_wheels.py already holds the resolved ITK commit on the host, so
it now reads that commit's committer time there and adds it to every
step's environment, native and container. A value already set by the
caller wins, through the shared build_wheels.source_date_epoch helper.
The container scripts forward the host value into the container.
The cold Windows release build ended with `sccache --show-stats`
reporting 0 compile requests although the cache grew. The server most
likely idled out (SCCACHE_IDLE_TIMEOUT, default 600 s) during a long
stretch without compiles, a later compile started a fresh server, and
the final statistics came from that one.

Set SCCACHE_IDLE_TIMEOUT=0 for sccache unless the caller set it. sccache
0.19 reads it when the server starts (src/server.rs, get_idle_timeout;
0 disables idle shutdown), and the session restarts the server after the
environment is set and stops it at the end.
The step report resumes a run by skipping every step already marked
done, whatever the run is building. A report from an earlier build of a
different ITK commit therefore makes a new run skip all of its steps.

BuildManager now takes an optional mapping of inputs, records it in the
report, and starts every step again as pending when the recorded inputs
differ or are missing. Identical inputs keep the resume behavior that
a rerun after a failed step relies on. changed_inputs names the inputs
that differed, so a caller can say why the steps run again.
Release build roots are fixed, so a second release-wheels run on a host
reads the step report left by the first. Every step was already done,
so a run for a new ITK commit skipped the ITK build and the wheel build
and reported the earlier run's wheels as its own.

The builder now records three inputs in the report: the commit checked
out in the ITK source (resolved, since a branch name builds a new commit
each day), the package version, and a fingerprint. The fingerprint
hashes pixi.lock, the ITKPythonPackage scripts and CMake files, the
builder's CMake definitions (which carry the compiler cache launcher),
the compiler cache switch, and the manylinux version and image. Any
change runs every step again in the kept tree and says which input
changed; identical inputs still resume after a failed step.
dist/ is never emptied and release build roots are fixed. The macOS
repair step runs delocate on every dist/*.whl, the import test installs
itk with pip -f dist, and the release summary lists every matching
wheel, so wheels left by an earlier run were repaired, tested, and
reported as the new run's.

Before the wheel step builds, the wheels at the top of dist/ now move to
dist/earlier-wheels/<YYYYMMDD_HHMMSS>/. Nothing is deleted, and the
globs and pip -f do not look into subdirectories. The move happens only
when the wheel step runs, so a rerun with identical inputs that resumes
after the wheel step keeps the wheels it built. The release summary's
listing is a function now, so a test can show it skips moved wheels.
Two Windows builds of the same source gave wheels whose .pyd and .dll
files, tbb12.dll among them, differed only in the PE COFF TimeDateStamp
and the debug-directory TimeDateStamp: link.exe writes the link time
there. SOURCE_DATE_EPOCH already fixes the zip timestamps, so these
fields were all that kept the wheels from being identical. /Brepro
makes the linker write a hash of the binary there instead.

build_wheels.py now adds /Brepro to the _LINK_ environment variable on
Windows, next to where it sets SOURCE_DATE_EPOCH, keeping any value the
caller set and not adding it twice. Microsoft's "MSVC linker reference"
(learn.microsoft.com/en-us/cpp/build/reference/linking, section "LINK
environment variables") states that the linker "appends the options and
arguments defined in the _LINK_ environment variable to the command
line arguments before processing". link.exe reads it itself, so it
reaches every link that CMake and Ninja run for ITK, the superbuild's
ExternalProjects, the wheels, and remote modules, with no CMake flag
changes and with CMake's default linker flags left intact. The
windows-build environment uses the conda-forge vs2022_win-64 toolchain,
which is MSVC link.exe. Linux and macOS are unchanged.
Replace the black hook with ruff-format, run after ruff-check, and drop
[tool.black]. ruff also formats code examples in docstrings and
Markdown, so blacken-docs is limited to the reStructuredText docs. The
reformatted lines are f-string spacing and quotes in scripts/ and the
quotes in one docstring example.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants